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Abstract 

Background: The DECIDE framework was developed to support evidence-informed health system decisions 
through evidence summaries tailored to health policymakers. The objective of this study was to determine 
policymakers' perceptions regarding the criteria in the DECIDE framework and how best to summarise and present 
evidence to support health system decisions. 

Methods: We conducted an online survey of a diverse group of stakeholders with health system decision 
experience from 15 countries and the World Health Organization. We asked about perceptions of criteria relevant to 
making health system decisions, use of evidence, grading systems, and evidence summaries. 

Results: We received 112 responses (70% response rate). Most respondents had healthcare (85%) and research 
(79%) experience. They (99%) indicated that systematic consideration of the available evidence would help to 
improve health system decision-making processes and supported the use of evidence from other countries (94%) 
and grading systems (81%). All ten criteria in the DECIDE framework were rated as important in the 
decision-making process. Respondents had divergent views regarding whether the same (38%) or different (45%) 
grading systems should be used across different types of health decisions. All components of our evidence 
summary were rated as important by over 90% of respondents. 

Conclusions: Survey respondents were supportive of the DECIDE framework for health system decisions and the 
use of succinct summaries of the estimated size of effects and the quality of evidence. It is uncertain whether the 
findings of this survey represent the views of policymakers with little or no healthcare and research experience. 
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Background 

Decisions regarding health systems are often poUtical 
processes involving a number of policymakers and other 
key stakeholders. There is a need to support these deci- 
sions with the best available evidence, however, the 
effects of health system interventions are often uncer- 
tain and stakeholders may have a low level of medical 
and research literacy. The Developing and Evaluating 
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Communication Strategies to Support Informed Decisions 
and Practice Based on Evidence (DECIDE) project is a 
collaborative research project funded by the European 
Commissions Seventh Framework Programme [www.de- 
cide-collaboration.eu]. The projects objective is to develop 
and evaluate communication strategies to support evidence- 
informed decisions by building on the work of the Grading 
of Recommendations Assessment, Development, and Evalu- 
ation (GRADE) Working Group [www.gradeworkinggroup. 
org] and the Cochrane Applicability and Recommenda- 
tions Methods Group [www.armg.cochrane.org]. As part 
of the DECIDE project, we have developed a framework 
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for communicating evidence to inform decisions about 
health systems [http://www.decide-collaboration.eu/WP5/ 
Strategies/Framework] . 

The development of this framework was informed by 
our earlier work on plain language summaries of system- 
atic reviews [1] and work on evidence-based policy briefs 
[2-4]. The framework includes relevant criteria for mak- 
ing health system decisions as well as evidence to inform 
judgements about each criterion. The criteria included 
in the framework are seriousness of the problem, num- 
ber of people affected, quality of the evidence, size of the 
benefits, size of the adverse effects, resource use (costs), 
value for money, impacts on equity, implementability 
(feasibility), and acceptability. These framework criteria 
emerged from a literature review, brainstorming, feed- 
back from stakeholders, and application of the frame- 
work to examples. This survey, along with user testing 
and further application of the framework to health sys- 
tem decisions, will further inform the selection of cri- 
teria included in the framework and how they are 
presented. 

There are several different systems available to grade 
the quality of evidence on the effects of healthcare inter- 
ventions. Most of these have been used primarily for 
clinical practice guidelines and those systems have be- 
come increasingly similar to GRADE or replaced by 
GRADE [5-7]. Other systems have been used for popula- 
tion health guidelines (primarily for public health rather 
than health system interventions), such as the systems 
used by the Task Force on Community Preventive Ser- 
vices [6,8,9], although the GRADE system is also used 
for public health and health system recommendations 
[10,11]. All of these grading systems have focused pri- 
marily on grading the quality of evidence and only to a 
lesser extent, if at all, on frameworks for going from evi- 
dence to public health or health system decisions or 
recommendations. 

The use of grading systems for rating the quality of 
evidence is seen as essential for evidence-based guideline 
development. The GRADE methodology has been 
adopted by WHO, as well as by national guideline devel- 
opers and others to assess the quality of evidence in 
guideline development as it enables a comprehensive, 
transparent and structured analysis of available litera- 
ture, whilst clearly communicating the quality of evi- 
dence and strength of the recommendations [1,12,13]. 
However, there is considerable debate as to whether the 
same approach to grading evidence for clinical decisions 
should also be used for health system decisions. Advo- 
cates for the use of the same grading system for both 
types of decisions state it can minimise confusion, re- 
duce the risk of bias, and maintain transparency and 
consistency across different types of decisions. Critics 
state that a rating system that is appropriate for clinical 



interventions may not discriminate evidence in a way 
that is appropriate for health system programmes or pol- 
icies, potentially disadvantaging effective interventions 
that are not amenable to randomized controlled trials [10]. 

The presentation of evidence to inform health system 
decisions requires the development of summaries that 
can 1) communicate complex information in plain lan- 
guage; 2) clearly present the anticipated effects and the 
certainty (quality) of the underlying evidence; and 3) ef- 
fectively convey uncertainty. We have previously devel- 
oped and user-tested evidence summaries tailored for 
health policymakers in low- and middle-income coun- 
tries [2], built on earlier findings for Cochrane reviews 
[14,15]. An evidence summary based on the DECIDE 
framework needs to clearly communicate evidence relat- 
ing to each of the ten DECIDE framework criteria, in- 
cluding the effect sizes and quality of evidence available 
for the anticipated desirable and undesirable effects of 
the intervention being considered. The design of these 
evidence summaries to support policymakers' under- 
standing of evidence therefore requires a detailed ana- 
lysis of their perceptions of certainty (quality) of the 
evidence and how it is assessed. 

To obtain a better understanding of policymakers', 
managers' and other stakeholders' perceptions of the cri- 
teria for health system decisions in the DECIDE frame- 
work, and how evidence on these should be presented, 
we conducted an international online survey. The survey 
focused on stakeholders' perceptions of each of the ten 
DECIDE framework criteria, the grading of health sys- 
tem evidence, and the use of evidence summaries. This 
paper aimed to collect information regarding the experi- 
ence and perceptions of participants with respect to the 
proposed criteria, assessments of the quality of evidence 
used to inform judgements about the effects of interven- 
tions, and summaries of research evidence of the effects 
of health system interventions. 

Methods 

We conducted an international online survey to deter- 
mine perceptions of the importance and use of the cri- 
teria within the DECIDE framework and evidence 
summaries. We aimed to survey a diverse (rather than 
representative) group of policymakers from different 
countries, with a wide range of experience with different 
types of health policy and management decisions and 
with different perspectives. Our sampling frame included 
the nine DECIDE partner countries (Canada, England 
and Wales, Finland, Germany, Italy, the Netherlands, 
Norway, Scotland, and Spain), the World Health 
Organization (WHO), and six SURE (Supporting the 
Use of Research Evidence for policy in African health 
systems) partner countries, namely Cameroon, Ethiopia, 
Mozambique, Zambia, Uganda, and South Africa [www. 
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who.int/evidence/sure/en/]. SURE is a related collabora- 
tive research project with the objective of developing 
and evaluating strategies for improving access to and use 
of research evidence in health policy development in 
Africa. These countries were included as we had suffi- 
cient networks linked to the DECIDE and SURE projects 
to provide the necessary referrals to potential partici- 
pants. Inclusion criteria for survey participants included 
people responsible for health system decisions and 
stakeholders with an interest in and experience with 
health system decisions, such as delivery, governance 
and financial arrangements, and strategies for 
implementing health system changes. This could include 
public officials, managers, health workers, and represen- 
tatives of special interest groups, international organiza- 
tions, non-governmental organisations, donor countries, 
or the general public. Our partners in the DECIDE and 
SURE projects from each country and other personal 
contacts helped us to identify 10 to 20 people who were 
not involved with the DECIDE collaboration in each of 
the 15 countries and within WHO, with the aim of 
obtaining 10 completed questionnaires from each. We 
prepared an online survey (in English only) using the 
LimeService online platform [https://www.limeservice. 
com/] (Additional file 1 Table SI). The survey was re- 
vised after pilot testing in a small group of policymakers 
who provided feedback on the content, length, clarity, 
and ease of use. Informed consent was obtained from 
survey participants prior to commencing the survey and 
results were de-identified when exported for analysis to 
protect confidentiality. 

To determine current use and perceived importance of 
the DECIDE criteria in health system decision-making, 
we asked participants to provide an example of a health 
system decision with which they had been involved or 
were familiar. Participants were asked to rate whether 
each of the ten criteria in the DECIDE framework (ser- 
iousness of the problem, number of people affected, 
quality of available evidence, benefits (desirable effects), 
adverse (undesirable) effects, resource use (cost), value 
for money, impacts on equity, implementability (feasibil- 
ity), and acceptability) had been considered as part of 
this healthcare decision on a 3-point scale (yes, no, un- 
sure). If a given criterion had not been considered, they 
were asked whether they considered it relevant, and also 
to state any other criteria they considered relevant. The 
opportunity to make optional comments was provided. 
Participants were also asked to rate the importance (im- 
portant, probably important, not sure, probably not im- 
portant, not important) of each of the ten criteria, with 
optional comments if desired. 

To determine perceptions on the use of evidence in 
health system decisions, we asked participants about 
their use of 1) evidence from other countries; 2) 



systematic reviews to inform their health system deci- 
sion; and 3) evidence grading systems. Specifically, we 
asked whether they believed that evidence grading sys- 
tems should be the same or different for health system 
decisions, compared to clinical decisions. We also 
assessed the perceptions of participants as to the con- 
tents of summaries of evidence. These summaries, based 
on the best available evidence, should be concise yet in- 
clude all of the key information needed to inform a 
health system decision. We asked participants to rate 
the importance of including certain types of information 
(components) in these summaries, namely: effect sizes, 
confidence intervals, numbers of studies, and the quality 
of the evidence. Participants also provided information 
on their research, healthcare, and decision-making 
experience. 

Participants were contacted by email and asked to 
complete the online survey. Initial contacts were made 
by our partners in each country or directly by us. Non- 
responders received reminders via email at two and four 
weeks after the initial invitation. We summarised the re- 
sults using frequencies and percentages, and collated 
provided comments. Our primary analysis focused on 
implications for our evidence to health system decisions 
framework and evidence summaries. We explored po- 
tential differences in responses across participants from 
different countries (DECIDE versus SURE partner coun- 
tries) and across groups with different types of experience 
(with versus without research training or experience). 

Results 

We received 112 responses (70% response rate) to the 
online survey. Of these, 84 responses (75%) were 
complete and 28 (25%) were partially complete. We re- 
ceived 23 responses (46% response rate) from the six 
SURE countries, 84 responses (93% response rate) from 
the nine DECIDE partner countries, and five responses 
(50% response rate) from WHO. Table 1 describes the 
characteristics of survey participants who provided back- 
ground information (n = 84). Most (84.5%) of the re- 
spondents had healthcare professional training or had 
worked as a healthcare professional. The majority of 
these were physicians (87.1%), and most (69.0%) had 
over ten years clinical experience. Most (78.6%) also had 
some form of research training or experience; 54.5% of 
these had over ten years of research experience. Respon- 
dents had worked in a variety of organizations, most fre- 
quently in national governments (63.1%) and public 
organizations (48.8%). The most common forms of health 
system decisions made or supported by respondents were 
decisions regarding the selection of healthcare policies, re- 
forms or programmes (77.4%), and decisions on their im- 
plementation (78.6%). 
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Table 1 Characteristics of survey participants 

Survey participants, n (%) 

Total (completed entire survey) 84 (75.0) 

No healthcare professional training or experience working as a healthcare professional 13 (15.5) 

Healthcare professional training or experience working as healthcare professional 71 (84.5) 

Physician 61 (85.9) 

Nurse 1 (1.4) 

Other 8(11.3) 

Missing 1 (1.4) 
Years of experience as healthcare professional 

I - 10 years 22 (31.0) 

II -20 years 18(25.4) 
21 -30 years 14(19.7) 
31 -40 years 14 (19.7) 
40+ years 3 (4.2) 

Research training or experience working as a researcher 66 (78.6) 

MSc or equivalent 24 (36.4) 

PhD or equivalent 40 (60.6) 

Other 2 (3.0) 

Years of experience as a researcher 

I - 10 years 30 (45.5) 

I I - 20 years 20 (30.3) 
21 -30 years 12 (18.2) 
31 -40 years 3 (4.5) 
40+ years 1 (1 .5) 

Levels of current and previous work* 

International governmental organization 15 (17.9) 

National government 53 (63.1) 

Regional government 26(31.0) 

Local government 18(21.4) 

Public organization 41 (48.8) 

Private organization 16 (19.1) 

Other 19(22.6) 

Types of health system decisions supported 

Experience representing the views of stakeholders in policy or management processes 46 (54.8) 

Decisions about healthcare policies, reforms or programmes 65 (77.4) 

Decisions about implementation of healthcare policies, reforms or programmes 66 (78.6) 

Management decisions about health system arrangements 44 (52.4) 

Decisions about quality improvement, patient safety or implementation of clinical guidelines 56 (66.7) 

Other 5 (6.0) 
*These categories were not mutually exclusive and hence the sum is greater than 100%. 

We asked participants to describe a recent healthcare considered for their health system decisions. Over 75% of re- 
decision with which they had been involved or had the spondents stated they had considered these criteria in their 
opportunity to follow closely. One hundred and two re- decision, except for value for money (67%) and impacts on 
spondents (91.1%) had such an example and 10 (8.9%) equity (70%). Comments were infrequent but almost univer- 
respondents imagined a current or recent decision. Figure 1 sally in support of the need to consider these criteria and 
shows their responses regarding whether our criteria were the lack of evidence on certain criteria, such as equity and 
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■ No and not relevant 

E No, and probably not relevant 
Unsure 

No and not sure if relevant 

■ No, but probably would have been relevant 

■ No, but would have been relevant 

■ Yes 



Figure 1 Survey responses regarding criteria considered for individual liealth system decisions. 



implementation, for health system decisions. Two respon- 
dents indicated that consideration of these criteria is 
mandatory for decisions tal<en in their organization. Most 
(86.4%) of the respondents had considered evidence from 
other countries for their decision, and 94.3% of respondents 
agreed or somewhat agreed that evidence from other coun- 
tries should be used to inform health system decisions. Most 
(90.9%) of the respondents stated they knew what a system- 
atic review was. However, only 60.2% of respondents had 
used evidence from a systematic review to inform their deci- 
sion. In response to our question on whether systematic 
consideration of the available research evidence would help 



to improve health system decision-making, 98.9% stated it 
would or it probably would. 

Respondent ratings on the importance of the ten cri- 
teria are summarised in Figure 2. Every criterion was 
rated as important or probably important by over 90% of 
respondents. Nearly all respondents (99.2%) agreed that 
explicit consideration of the ten criteria would help or 
probably help to improve health system decision-making. 
While comments supported the importance of the criteria, 
several respondents identified further criteria. These 
included sustainability of implemented changes, post- 
implementation monitoring systems, human resource 
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implications, and "environmental" considerations (assessing 
what is occurring in nearby areas or similar jurisdictions). 

Most respondents agreed that a system of grading evi- 
dence would either definitely (59.7%) or probably 
(33.3%) improve health system decision-making pro- 
cesses (Table 2). When asked whether such grading sys- 
tems should be the same for clinical and health system 
decisions, 38.4% said it should or probably should be the 
same, 45.3% indicated that different grading systems 
should or probably should be used for clinical and health 
system decisions, and 16.3% were neutral. This disagree- 
ment was also reflected in comments. Some respondents 
stated that using the same grading system improves 
consistency, transparency, and reproducibility, while 
others stated that they are fundamentally different types 
of decisions and evidence, requiring different grading 
systems. Two respondents (both from DECIDE partner 
countries) indicated an overall dislike of evidence grad- 
ing systems. Respondent ratings on the importance of 
components of a summary of evidence are described in 
Figure 3. There was general agreement that all of the 
summary components were important or probably im- 
portant, ranging from 79% (confidence intervals for ef- 
fect estimates) to 96% (description of the quality of the 
evidence). All comments were in broad support, includ- 
ing several highlighting the need for clarity, simplicity, 
and brevity. Others suggested the use of disability- 
adjusted life years (DALYs) and number needed to treat 
(NNT) as important measures of effect. 

Overall, there was relatively little variation in the re- 
sponses. Nonetheless, we explored potential differences 
in responses across participants from different countries 
and across groups with different types of research ex- 
perience (Additional file 1 Table SI). We did not find 



any apparent differences in responses based on the re- 
spondents' country (DECIDE versus SURE partner coun- 
tries) or experience (with compared to without research 
training or experience). 

Discussion 

We received 112 responses to an online survey from a 
diverse, international group of policymakers, managers, 
their support staff and other stakeholders to better 
understand their perceptions of the ten criteria in the 
DECIDE framework for going from evidence to health 
system decisions. We also obtained their perceptions of 
the components of summaries of evidence. Our respon- 
dents had a high level of professional healthcare and re- 
search experience and training and had worked in a 
wide range of levels and organizations. This may be due 
to the way in which they were identified, or a greater 
interest among those policymakers with clinical or re- 
search experience to participate in the survey. Our 
sample of policymakers was not intended to be represen- 
tative of included countries, but was rather to obtain a 
diverse range of views from respondents. 

Respondents had generally considered all ten criteria 
in the DECIDE framework in their own healthcare deci- 
sions and consistently rated these criteria as important 
or probably important to decision-making. In earlier 
work, Guindo and colleagues identified a range of 
healthcare decision criteria and criteria-based decision- 
making tools used in empirical studies of health system 
decisions - the most frequently cited criteria were equity 
and fairness, efficacy/effectiveness, stakeholder interests 
and pressures, cost effectiveness, strength of evidence, 
and safety [16]. In our survey, comments by respondents 
indicated that not only were similar criteria considered 



Table 2 Respondents' views on the use of grading systems to rate the quality of evidence of health decisions 

Survey participants, 

n (%) 

A system of grading the quality of evidence can help improve health system decision-making processes. 

Yes 

Probably 
Not sure 
Probably not 



No 



52 (59.7) 
29 (33.3) 
5 (5.8) 
1 (1.2) 
0 (0.0) 



Do you think that a system for rating the quality of evidence should be consistent for different types of decisions or that there should be 
different systems for different types of decisions (e.g., for clinical decisions and for health system decisions)? 

The SAME SYSTEM DEFINITELY should be used for rating the quality of evidence for clinical and health system decisions 9 (10.5) 

The SAME SYSTEM PROBABLY should be used for rating the quality of evidence for clinical and health system decisions 24 (27.9) 

Neutral 14(16.3) 

A DIFFERENT SYSTEM PROBABLY should be used for rating the quality of evidence for health system decisions than for 27 (31 .4) 
clinical decisions 

A DIFFERENT SYSTEM DEFINITELY should be used for rating the quality of evidence for health system decisions than for 12 (13.9) 
clinical decisions 



Vogel et al. Health Research Policy and Systems 201 3, 1 1 :1 9 
http://www.health-policy-systenns.conn/content/1 1/1/19 



Page 7 of 9 



90% 




>^ I don't know what that is 

■ Not important 
Probably not important 
Not sure 

Probably important 

■ Important 



Description of the size of Confidence intervals for Description size of effect Description of how Description of the 

the effect effect estimates in words much evidence quality of the evidence 

Figure 3 Survey responses rating the importance of components of a summary of evidence. 



important to decision-making processes, but that in 
some organizations, their consideration is mandatory. 
However, only 60.2% reported using systematic reviews 
to inform decision making. This may reflect both a lack 
of systematic reviews addressing relevant questions, as 
well as other reasons for not using research evidence to 
inform health policy decisions, as reported elsewhere 
[17,18]. 

Most respondents agreed that a system of grading evi- 
dence would improve health system decisions; however, 
there was considerable disagreement as to whether these 
grading systems should be uniform across clinical and 
health system decisions. This is consistent with the on- 
going debate in the literature on the application of the 
GRADE approach to a range of decisions [10,12]. While 
a common approach to grading evidence may reduce 
confusion, minimise conflicts of interest and enhance in- 
tellectual rigor, others have argued that this approach 
can be overly complex and can favour false negative con- 
clusions. Additionally, interventions that are not amen- 
able to randomised trials could be disadvantaged in 
terms of prioritisation, funding and implementation [12]. 
Criticisms made by our respondents of a uniform grad- 
ing approach concur with the literature that there is a 
lack of available high-quality evidence on health system 
interventions. Two respondents also indicated an overall 
dislike of grading systems, citing a tendency to oversim- 
plify complex issues or a lack of institutional capacity to 
provide training or support grading activities. However, 
the application of some form of systematic consideration 
of evidence to support health system decisions had 



broad support in our survey. Successful implementation 
of the GRADE approach for health system decision- 
making will need to address concerns about its applic- 
ability to health system evidence. 

Effectively communicating complex information through 
summaries has been proven in the use of the GRADE sum- 
mary of findings table [13,19] and SUPPORT summaries [2]. 
SUPPORT summaries summarise the best available evidence 
of the effects of health system interventions for low and 
middle-income countries. Such summaries may be particu- 
larly useful for policymakers without a strong health or re- 
search background. Although our respondents had a 
higher-than-expected level of healthcare and research ex- 
perience, their responses clearly indicated that all pro- 
posed components of the summaries of evidence have 
practical applications in health system decision processes. 
Comments strongly favoured clear, concise summaries in 
simple language, suggesting that researchers should con- 
sider ease of interpretation by policymakers with limited 
scientific literacy when preparing summaries. 

The strengths of this survey were a good response rate 
from a diverse range of countries, backgrounds, levels of 
decision-making, and organisations. The fact that 25% of 
the surveys were partially completed may have been due 
to survey length. The comparatively lower response rate 
from SURE partner countries means results are likely 
biased towards higher-resource settings, limiting their 
applicability to resource-constrained settings. One sig- 
nificant limitation of the survey was the relatively high 
level of healthcare and research experience amongst re- 
spondents; only four respondents had no healthcare or 
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research experience. This is probably not representative 
of health policymakers in general and potentially limits 
the generalizability of our findings. Nonetheless, the fact 
that the criteria included in the DECIDE framework 
were regarded as important for decision-making by these 
respondents provides further support for their inclusion 
in a framework for going from evidence to health system 
decisions. 

This survey confirmed the relevance of the criteria that 
we had identified and incorporated in the DECIDE frame- 
work for health system decisions and suggests that the 
framework is likely to be helpful for informing health sys- 
tem decisions. Further development and evaluation of the 
framework will be based on practical applications of the 
framework to health system and population health deci- 
sions and user testing [20]. Facets of the framework that 
will be addressed by user testing were adapted from the 
work by Morville et al. [21] and Rosenbaum et al. [22], 
and include 1) findability: can users locate what they are 
looking for?; 2) usefulness: does the framework have prac- 
tical value for the user?; 3) usability: how easy and satisfy- 
ing is the framework to use?; 4) understandability: do 
users understand the framework and the content cor- 
rectly?; 5) credibility: is this framework/content trust- 
worthy?; 6) desirability: is the framework something the 
user wants/responds positively to?; and 7) identification: 
does the framework feel like it was designed for "someone 
like me (the user)"? Further work will address the advan- 
tages and disadvantages of using the same versus different 
systems for grading evidence, further clarification of the 
included criteria, the need for additional criteria, and the 
perceptions of policymakers and stakeholders who do not 
have a research or health professional background. 

Conclusions 

Health system decision-making requires careful consid- 
eration of a multitude of variables, such as the magni- 
tude of the problem, the size of benefits and adverse 
effects, feasibility and acceptability, as well as resource 
and equity implications. Surveyed individuals supported 
the use of systems to grade the quality of evidence for 
health system decisions, but there was disagreement as 
to whether uniform or different grading systems should 
be used for health system and clinical decisions. Com- 
munication of evidence to policymakers and stake- 
holders involved in health system decisions should 
employ succinct summaries of measures of effect and 
the quality of evidence in clear and simple language. 

Additional file 



Additional file 1: Table 51. Participant responses on quality of 
evidence and components of evidence summaries, stratified by research 
experience and country of origin. Differences in responses by respondent 



research experience and country of origin were tested using tests; 
none were significant at the P <0.05 level. 
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